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ABSTRACT 

This  thesis  was  a  study  which  deals  with  two  basics  concepts  in  human 
decision  making.  The  first  is  the  role  of  information  relevance,  specifically  the 
adverse  effects  of  irrelevant  information  on  decision  quality.  The  second  key 
concept  was  cognitive  feedback  and  its  value  for  supporting  decision  making.  The 
thesis  was  designed  to  research  the  effectiveness  of  cognitive  feedback  in  reducing 
the  adverse  effects  of  irrelevant  information.  The  experiment  tested  the  Lens 
Model  indices:  achievement,  consistency  and  matching  in  task  conditions  of  high 
and  low  predictability.  Subjects  were  divided  into  blocks  which  differed  in  the 
availability  of  cognitive  feedback  and  predictability  conditions.  The  results  of  the 
experiment  showed  the  subjects  performed  better  in  all  Lens  Model  indices  in  the 
cognitive  feedback  condition.  Subjects  also  had  superior  performance  across  all 
Lens  Model  indices  in  the  high  predictability  condition.  This  thesis  was  intended 
to  contribute  to  the  research  in  the  subject  of  human  decision  making.  The  results 
were  of  importance  in  support  of  future  design  of  decision  support  systems. 
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I .   Introduction 


A.   Research  Context 

1 .  Use  of  feedback  on  decision  making 

Researchers  in  several  fields  have  studied  the  effects 
of  feedback  on  decision  making.  From  the  fields  of  psychology 
and  psychiatry,  research  has  studied  the  way  in  which 
cognitive  feedback  is  used  by  an  individual  to  understand 
his/her  environment  (Doherty  and  Balzer,  1988) .  Researchers 
in  the  field  of  marketing  have  been  interested  in  how  "good" 
decision  makers  depend  on  the  use  of  cognitive  feedback 
(Jacoby,  Mazursky,  Troutman,  and  Kuss,  1984)  .  Research  has 
shown  that  some  forms  of  feedback  can  have  an  effect  on  the 
quality  of  decision  making.  From  this  research,  and  others, 
there  have  emerged  several  theories  which  support  this  thesis . 
These  topics  will  be  discussed  in  detail  in  Chapter  II. 

2 .  Effects   of   irrelevant   information   in   decision 
making 

The  effect  of  irrelevant  information  on  decision 

making  has  been  a  subject  of  continuing  attention  in  the  field 

of  social  and  clinical  psychology.  Research  has  been  directed 

at  the  effects  of  information  relevance  on  decision  making 

(Streufert  S.,   1973),   reducing  the  effects  of  irrelevant 


information  on  experienced  decision  makers  (Gaeth  and 
Shanteau,  1984) ,  and  the  perception  of  information  relevance 
(Streufert  and  Streufert,  1970) .  One  well  accepted  theory 
concerning  information  relevance  is  that  the  presence  of 
irrelevant  information  is  detrimental  to  decision  making.  The 
concept  of  information  relevance  is  of  key  importance  to  this 
thesis .  Research  and  theories  pertaining  to  information 
relevance  will  be  discussed  in  Chapter  II. 

The  research  proposed  here  addresses  the  two  points 
made  above  -  (a)  given  that  the  presence  of  irrelevant 
information  has  an  effect  on  decision  making,  (b)  can  the  use 
of  feedback  reduce  the  effects  of  irrelevant  information.  In 
this  study  we  will  examine  the  effects  of  irrelevant 
information  in  task  environments  characterized  by  different 
degrees  of  predictability.  We  will  draw  conclusions  as  to  the 
effectiveness  of  feedback  as  a  method  of  reducing  the  effects 
of  irrelevant  information. 

3 .   Relevance  to  information  systems  research 

The  relevance  of  this  study  to  research  to  Information 
Systems  (IS)  is  in  the  domain  of  Decision  Support  Systems 
(DSS) .  With  the  rapid  growth  of  demand  for  both  specialized 
and  off-the-shelf  DSS,  emphasis  is  being  placed  on  the  quality 
of  design.  DSS  provide  users  with  information  in  order  to 
enhance  the  quality  of  their  decisions.  Not  all  information 
displayed  to  the  user  at  one  time  is  relevant.    If  the 


decision  maker  is  given  the  opportunity  to  study  his/her  own 
decision  rules  through  the  use  of  some  form  of  cognitive 
feedback,  he  can  improve  his/her  decision  rules.  Not  all  DSS 
of  today  provide  CFB  to  the  user.  This  research  will  help  to 
determine  if  such  a  capability  can  improve  the  quality  of  the 
DSS. 

B.   Problem  domain 

In  this  section,  we  outline  the  elements  of  the  problem 
domain,  thereby  constituting  the  basis  for  the  research 
question  -  the  notion  of  cognitive  feedback  as  an  aid  to 
decision  making  in  environments  characterized  by  the  presence 
of  irrelevant  information. 

1 .  Irrelevant  information 

The  effects  of  irrelevant  information  on  decision 
making  have  been  studied  extensively.  The  consensus  from  this 
research  yields  two  basic  premises  that  are  essential  to  this 
study.  First,  the  presence  of  irrelevant  information  is  a 
detriment  to  decision  making  (Gaeth  and  Shanteau,  1984)  . 
Second,  the  adverse  effects  of  irrelevant  information  are 
greater  in  complex  environments  than  in  relatively  simple 
environments  (Streufert  and  Streufert,  1973) . 

2 .  Cognitive  feedback 

The  term  feedback  describes  an  environment  that 
returns  some  measure  of  the  output  of  a  system  back  to  the 
system  which  produced  the  output.   Feedback  then  allows  a 


person  to  compare  his/her  present  state  to  an  ideal  state,  to 
adjust  itself  in  light  of  that  comparison  (Doherty  and  Balzer, 
1988)  .  Feedback  can  be  provided  in  two  forms  -  outcome  and 
cognitive.  Outcome  feedback  deals  with  the  accuracy  of  a 
response.  Cognitive  feedback  provides  information  as  to  how 
that  response  was  generated.  Cognitive  feedback  is  the  return 
of  some  measure  of  the  output  of  a  person's  cognitive 
processes,  to  help  that  person  come  to  terms  with  his/her 
environment . 

Cognitive  feedback  can  take  three  forms  in  an 
experimental  setting: 

(a)  information  about  the  relationships  between  cues 
and  criterion,  i.e.  information  about  the  task: 

(b)  information  about  relationships  between  cues  and 
the  person's  inference,  i.e.  information  about  the  person's 
cognitive  state,  sometimes  referred  to  as  insight; 

(c)  information  about  relationships  between 
cognitions  and  distal  objects.  This  category  comprises 
indices  of  "functional  validity"  information  (Doherty  and 
Balzar,  1988)  . 

3 .   Research  question 

Given  that  the  presence  of  irrelevant  information 
adversely  affects  task  performance,  can  the  use  of  cognitive 
feedback  improve  the  quality  of  decisions? 


C.   Organization  of  the  thesis 

The  discussion  proceeds  as  follows.  Chapter  II  reviews 
the  research  in  the  effects  of  irrelevant  information  and  the 
use  of  feedback  in  task  situations.  Chapter  III  describes  the 
experimental  setting  and  chapter  IV  discusses  the  data 
analysis  and  results.  Chapter  V  summarizes  and  draws 
conclusions . 


II .    RESEARCH 


A.   The  Research  Question  -  A  Conceptual  Framework 

The  research  question  deals  with  two  basic  theoretical 
areas:  information  relevance  and  cognitive  feedback.  The 
purpose  of  this  chapter  is  to  lay  the  groundwork  in  these  two 
research  areas,  giving  logical  support  for  the  hypotheses  to 
be  made  later  in  Chapter  III.  Important  to  both  of  these 
research  areas  is  the  topic  of  human  judgment. 

1 .  Human  Judgment 

Understanding  human  judgment  is  important  in  the 
context  of  this  research.  Changes  in  human  judgment  are  what 
we  hope  to  effect  by  manipulation  of  the  dependent  variables . 
A  background  discussion  of  human  judgment  theories  is  included 
in  section  A.l.  The  mathematical  and  experimental 
representation  of  human  judgment  is  provided  by  the  lens  model 
(Dudycha  and  Naylor,  1966)  .  The  lens  model  and  issues 
surrounding  linear  models  of  human  judgment  will  be  discussed 
in  section  B.l. 

2 .  Information  Relevance 

Research  in  the  area  of  information  relevance  is 
extensive  (Streufert,  1973;  Gaeth  and  Shanteau,  1981,  1984; 
Adelman,   1981) .    It  is  commonly  accepted  that  irrelevant 


information  serves  as  a  detriment  to  decision  making.   The 
effects   of   information   relevance   vary   under   different 
circumstances.     Section  C  will  discuss  the  elements  of 
information  relevance  as  it  pertains  to  this  research. 
3 .   The  Research  Problem 

The  research  problem  can  be  framed  by  the  following 
statements : 

(i)    Individuals  make  models  of  decision  processes. 

(ii)  Irrelevant  information  serves  as  a  detriment  to 
successful  development  and  use  of  a  decision  model . 

(iii)  Cognitive  feedback  has  been  proven  successful  in 
improving  decision  making,  by  providing  decision  makers  a 
better  insight  into  their  model  development  and  usage 
strategy. 

(iv)  So,  given  that  cognitive  feedback  improves 
decision  making,  can  it  be  useful  in  overcoming  the 
detrimental  effects  of  irrelevant  information? 

B .   Human  Judgment 

According  to  Brehmer,  knowledge  is  a  relation  between  two 
systems.  One  system  in  this  relationship  is,  for  the  purpose 
of  studying  human  judgment,  a  person  and  the  other  is  some 
portion  of  the  environment.  A  person  who  is  believed  to 
"know"  a  great  deal  is  someone  who  understands  relationships 
with  his/her  environment  better  that  someone  who  understands 
these  relationships  less.  These  relationships  are  believed  to 


be  probabalistic .  Thus,  perfect  knowledge  is  infeasible, 
only  knowledge  where  the  person  has  a  high  probability  that 
what  he  believes  he  knows,  is  actually  the  case. 

It  has  been  the  focus  of  many  researchers  in  experimental 
and  clinical  psychology  to  study  the  way  in  which  humans  seek 
relationships  between  themselves  and  their  environment 
(Brunswik,  Brehmer,  Doherty  and  Balzer,  et  al) .  Doherty  and 
Balzer  referred  to  knowledge  representation  as  a  complex  set 
of  relations  called  a  policy.  Policies  are  sampled  from  the 
stimuli  and  then  analyzed  for  their  relevance.  The  relevant 
components  are  then  tried  for  their  usefulness,  either  one  by 
one  or  in  some  combination  (Doherty  and  Balzer,  1980) . 

The  use  of  linear  models  was  first  suggested  by  Brunswik 
(1944)  for  studies  of  perception.  A  linear  model  suggests 
that  there  is  a  probablistic  and  functional  relationship 
between  an  individual  and  his  environment,  and  that  the 
functions  can  be  described  and  measured.  These  methods  are 
used  to  test  a  series  of  hypotheses  about  the  nature  of  the 
judgment  process,  hypotheses  about  the  nature  of  cue  weights, 
function  forms,  combination  rules,  and  predictability 
(Brehmer,  1979)  .  The  name  of  the  model  developed  by  Brunswik 
was  the  Lens  Model . 

1 .   The  Lens  Model 

The  Lens  Model  is  a  linear  regression  model  developed 
by  Brunswik  for  assessing  the  dynamics  of  human  choice 


behavior  in  a  probablistic  environment.   The  following  text 

from  Dudycha  and  Naylor  describes  well  the  components  of  the 

Lens  Model: 

The  three  basic  elements  to  the  model  are  the  cues  or 
stimulus  dimensions  (X1  ....  Xk) ,  the  correct  response  or 
answer  (Y.) ,  and  the  observed  response  of  the  individual 
(Y.) .  Any  choice  or  decision  situation  (trial)  must  of 
necessity  include  these  three  elements .  Given  many  such 
decision  trials  it  is  possible  to  determine  the 
statistical  relevance  of  a  cue  X±  as  a  predictor  of  the 
criterion  Y.  by  computing  the  zero-order  correlation  r#i 
over  trials.  This  true  cue  validity,  when  squared,  can  be 
interpreted  as  an  index  of  the  diagnostic  power  of  that 
cue  as  a  source  of  information  for  predicting  the  correct 
state  of  nature. 

Also  from  Dudycha  and  Naylor  are  the  following  values 
used  in  the  model : 

r9  =  the  correlation  between  the  true  or  observed 
criterion  and  the  predicted  criterion. 

rB  =  the  index  of  subject  consistency,  or  the  degree 
to  which  the  subject  consistently  utilized  his  strategy  as 
defined  by  his  multiple  regression  equation. 

rn  =  the  correlation  which  denotes  subject 
achievement .  It  measures  the  degree  of  agreement  between  the 
criterion  values  and  responses  of  the  subject  over  n 
observations . 

rm  =  the  correlation  between  the  two  sets  of  n 
predicted  values,  which  reflects  the  degree  to  which  the 
regression  (policy)  equation  of  the  subject  "matches"  the 
regression  (optimal)  equation  of  the  ecology. 

re.  =  the  correlation  which  reflects  the  degree  that 


the  optimal  equation  of  the  environment  can  predict  the  actual 
responses  of  the  subject. 

rc.  =  the  correlation  which  reflects  the  degree  to 
which  the  policy  equation  of  the  subject  can  predict  the  true 
criterion.  (The  above  variables  and  definitions  taken  from 
Dudycha  and  Nay lor,  1966) . 

The  Lens  Model  is  an  accepted  method  for  interpreting 
subject  responses  in  single  or  multiple  cue  probablistic 
learning  environments  with  linear  tasks.  Tucker  further 
developed  the  model  for  tasks  in  which  the  judgment  is  a 
linear  additive  function  to  yield  the  following  relationship: 

ra  =  GR.R. 
R„  represents  the  predictability  of  the  environment,  R, 
represents  the  consistency  of  the  subject,  and  G  is  the 
correlation  between  the  predictions  derived  from  the  linear 
model  of  the  environment  with  the  linear  model  of  the  subject 
(matching  index) .   Thus,  ra  is  called  the  achievement  index. 

C.   Information  Relevance 

Relevant  information  is  viewed  as  any  information  to  which 
a  meaningful  task-oriented  response  is  possible  (Streufert, 
1973)  .  Thus,  information  which  is  not  useful  in  the 
generation  of  meaningful  responses  is  irrelevant.  In  an 
experimental  setting,  as  well  as  in  nature,  irrelevant 
information  is  often  mixed  with  relevant  information. 

In  terms  of  the  lens  model,  irrelevant  information  is  a 
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cue  which  has  a  low  r#i .  A  cue  with  little  or  no  diagnostic 
power  (diagnostic  power  is  described  in  the  previous  section) 
is  mostly  or  entirely  irrelevant. 

Study  in  information  relevance  has  been  extensive.  Most 
data  suggest  that  increases  in  irrelevant  information  decrease 
performance  to  criterion  (Streufert,  1973;  Streufert  and 
Streufert, 1971;  Gaeth  and  Shanteau,  1984;  Ettenson  and 
Shanteau,  1987;  et  al)  .  This  premise  is  well  accepted  as  a 
starting  point  for  further  research  in  the  area  of  information 
relevance . 

Gaeth  and  Shanteau  (1984)  sought  to  discover  if  this 
influence  extended  to  experienced  decision  makers.  In  their 
study,  the  decision  makers  were  trained  agricultural  students. 
They  were  asked  to  describe  soil  texture.  Irrelevant  to  this 
determination  is  moisture  in  the  soil.  One  important  result 
of  the  experiment  is  benefit  of  training  in  reducing  the 
adverse  effects  of  irrelevant  information.  If  this  influence 
extends  to  experienced  decision  makers,  then  one  reasonable 
approach  to  improving  judgmental  skills  would  be  to  reduce  the 
effects  of  irrelevant  information  (Gaeth  and  Shanteau,  1984) . 

1 .   Effects  of  Irrelevant  Information 

The  effects  of  irrelevant  information  are  known  to 
vary  in  different  task  environments.  For  example,  adding 
irrelevant  information  in  simpler  environments  usually  does 
not  have  as  great  an  effect  in  producing  decrements  in  task 
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performance  (Streufert  and  Streufert,  1970) .  A  subject  can 
perform  well  in  a  task  environment  where  irrelevant 
information  is  present  as  long  as  he  can  sort  out  relevant  and 
irrelevant  information.  In  complex  environments  it  is  more 
difficult  to  differentiate  between  relevant  and  irrelevant 
information . 

Streufert  (1973)  performed  a  study  demonstrating  the 
decreases  in  performance  levels  previously  believed  to  be  the 
effect  of  load  increases  were  actually  a  function  of 
information  relevance. 

2 .   Effectiveness  of  Training 

Gaeth  and  Shanteau' s  research  using  training  is  of 
hallmark  importance  to  research  in  information  relevance.  The 
study  utilized  experienced  agricultural  judges  in  a  soil 
sampling  task.  In  a  pretest ,  irrelevant  information  was  shown 
to  influence  the  decisions  of  the  judges.  This  pretest  is  of 
some  consequence,  for  it  shows  that  even  experts  are  subject 
to  the  effects  of  information  irrelevance.  Subsequent 
training,  in  two  different  forms:  lecture  training  and 
interactive  training,  was  shown  to  improve  accuracy  and  reduce 
the  influence  of  irrelevance.  Gaeth  and  Shanteau' s  research 
utilizing  training  is  of  hallmark  importance  as  it  is  the 
first  substantial  work  investigating  the  potential  for 
reducing  the  adverse  effects  of  irrelevant  information. 
Additionally,  it  is  of  great  significance  that  the  study  was 
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able  to  differentiate  between  improvements  in  accuracy  and 
decreases  in  the  effects  of  irrelevant  information,  proving 
that  the  two  are  mutually  exclusive.  A  follow-up  study  showed 
that  the  benefit  from  training  was  shown  to  extend  for  a  year 
after  the  training  was  held. 

D .   Feedback 

1 .   Outcome  Feedback 

The  procedure  of  informing  the  subject  in  an 
experiment  of  the  correct  value  in  a  task  situation  (Y.) 
immediately  after  that  subject  produces  a  response  (Y.) 
defines  outcome  feedback  (OFB)  (Doherty  and  Balzer, 
1988) .  Research  has  attempted  to  correlate  improvements  in 
decision  quality  based  of  the  use  of  OFB.  It  is  commonly 
accepted  that  OFB  alone  is  of  little  assistance  to  decision 
makers  learning  complex  inference  tasks .  OFB  can  in  some 
cases  serve  as  a  detriment  to  decision  making  because  it 
encourages  departure  from  linearity  (Lindell,  1976) .  This 
departure  from  linearity  is  especially  likely  when  the  task 
environment  is  less  than  complete  predictability. 

Brehmer  (1980)  explains  the  reasons  why  OFB  is  not 

useful  for  enhancing  decision  making: 

Confirmation  will,  of  course,  not  teach  the  subjects  about 
the  actual  validity  of  their  hypothesis;  it  will  only  tell 
which  hypotheses  work,  although  the  reason  why  the 
hypotheses  work  may  be  very  different  from  what  the 
subject  thinks.  .  .  When  we  have  to  learn  from  outcomes,  it 
may  in  fact,  be  almost  impossible  to  discover  that  one 
really  does  not  know  anything.   This  is  especially  true 
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when  the  concepts  are  very  complex  in  the  sense  that  each 
instance  contains  many  dimensions. 

One  effect  seen  when  OFB  is  used  alone  is  when  a 
subject  is  given  OFB  demonstrating  that  his  response  is 
incorrect,  he  will  often  abandon  the  policy  which  was 
employed  to  produce  the  response,  even  if  it  was  the  correct 
policy.  Repeated  failure  to  get  the  "right"  answer  may  cause 
the  subject  to  abandon  the  idea  that  there  is  a  policy  at  all 
and  he  may  resort  to  guessing  (Doherty  and  Balzer,  1988)  . 

Outcome  feedback  in  a  less  than  perfectly  predictable 
task  apparently  acts  to  confuse  subjects  and  lower  their 
consistency.  Achievement  is  also  adversely  affected,  because 
of  this  lowered  consistency  (Schmitt,  Coyle,  and  Saari,  1977) . 
It  follows  that  outcome  feedback  in  task  environments  of  low 
predictability  is  not  useful  for  policy  revision. 
2 .   Defining  cognitive  feedback 

Cognitive  feedback  is  described  as  the  return  of  some 
measure  of  the  output  of  a  person's  cognitive  processes,  to 
help  that  person  come  to  terms  with  the  environment . 
Cognitive  feedback  provides  the  person  with  information 
describing  the  relationships: 

(a)  between  cues  and  the  criterion  (Task  Information 
TI)  ; 

(b)  between  cues  and  the  person's  inference,  i.e. 
information  about  the  person's  policy  (Cognitive  Information; 
CI)  and/or 
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(c)  between  cognitions  and  the  distal  objects.  This 
category  comprises  indices  of  functional  validity  (Doherty  and 
Balzer,1988)  .  The  third  category,  above,  is  added  to  parallel 
the  lens  model  accurately.  In  the  lens  model,  TI  includes  the 
indices  on  the  ecological  side  of  the  lens.  CI  includes  all 
information  on  the  subjects  side.  Functional  validity  refers 
to  all  three  measures  r„  G,  and  C  described  in  section  B.l. 

In  terms  of  the  Lens  Model,  TI  is  represented  by  R. 
or  R,2,  ecological  validities  (earlier  called  r#i  or  diagnostic 
power,  see  section  B.l)  or  function  forms  relating  the 
criterion  to  the  cues .  CI  is  represented  by  the  values  R.  or 
R.2  in  the  Lens  Model  as  correlation  indices  of  predictability 
of  the  subject  (consistency)  .  Another  form  of  CI  are  the 
usage  coefficients  (e.g.,  ris;  the  subject's  decision  policy) 
and  function  forms  relating  the  judgment  to  the  cues . 
3 .   Effectiveness  of  Cognitive  Feedback 

Cognitive  feedback  has  been  used  in  numerous 
experiments  to  produce  changes  in  decision  making  policy. 
Cognitively  oriented  feedback  results  in  higher  levels  of 
achievement  than  outcome  feedback  over  different  task 
properties  (Adelman,  1981)  .  Other  studies  have  demonstrated 
the  superiority  of  CFB  to  no  information,  as  evidenced  by 
policy  change  between  blocks  after  providing  CFB.  Research 
has  been  conducted  to  determine  which  component  of  CFB  (TI, 
CI,  or  FVI)  is  most  effective  in  making  change.    The  bulk  of 
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research  on  the  subject  shows  that  TI  alone  is  sufficient  to 
facilitate  change.  There  has  been  research  investigating  the 
use  of  CI  alone,  and  TI  +  CI .  There  is  not  adequate  research 
to  make  assumptions  about  differential  effects.  All  lens 
model  indices  seem  to  be  influenced  by  TI  and  TI  +  CI  with  R. 
being  more  sensitive  (Doherty  and  Balzer,  1988) . 

The  use  of  CFB  has  been  found  to  be  related  to  task 
predictability.  The  use  cognitive  feedback  under  conditions 
of  extremely  high  task  congruence  and  predictability  has  not 
been  shown  to  produce  higher  accuracy  than  outcome  feedback 
alone.  Schmitt  et  al .  (1977)  found  that  as  task 
predictability  decreased,  achievement  with  cognitive  feedback 
became  significantly  higher  than  that  with  outcome  feedback 
(Adelman,  1981) .  Predictability  is  varied  in  experimental 
settings  to  manipulate  task  complexity  (Steinman,  1976) . 

One  finding  of  double-systems  studies  pertains  to  the 
effects  of  task  predictability  on  the  cognitive  systems  of  the 
subjects .  Results  obtained  in  a  variety  of  circumstances  show 
that  the  consistency  of  a  cognitive  system  varies  with  the 
redictability  of  the  task,  the  lower  the  consistency  of  the 
cognitive  system  (Brehmer  1979) .  This  effect  is  related  to 
the  theory  that  policy  formation  and  utilization  is 
probablistic . 
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E.   Framing  the  Research  Question 

The  previous  sections  are  valuable  for  demonstrating  the 
need  for  and  validity  of  research  in  the  proposed  area.  In 
the  section  on  irrelevant  information  it  was  shown  that  a 
measurable  portion  of  difficulty  experienced  by  decision 
makers  in  complex  tasks  was  due  to  the  effects  of  irrelevant 
information.  Streufert  went  one  step  further  in  showing  that 
similar  losses  in  accuracy  previously  blamed  on  load  were 
actually  a  result  of  irrelevant  information.  Some  research 
has  begun  to  discover  means  for  recouping  or  avoiding  these 
losses.  Gaeth  and  Shanteau  have  had  success  using  training  as 
a  means  to  this  end.  They  further  point  out  that  research  in 
irrelevant  information  is  abundant,  while  research  into 
correcting  for  it  is  overdue  (Gaeth  and  Shanteau,  1981)  . 

The  use  of  cognitive  feedback  has  been  proven  to  be  an 
effective  means  of  improving  performance  in  many  different 
task  oriented  environments .  The  bulk  of  the  research 
indicates  that  the  more  complex  and  predictable  the  task  is, 
the  more  profound  the  benefit  is  when  cognitive  feedback  is 
utilized.  It  is  reasonable,  then,  to  believe  that  it  will 
also  have  a  measurable  effect  on  decision  making  in  task 
environments  subject  to  the  presence  of  irrelevant 
information. 

These  are  the  fundamental  principles  of  the  study.  It  is 
clear  from  these  themes  that  not  only  is  the  theoretical 
foundation  for  the  study  valid,  but  there  is  evidence  that  the 
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results  of  the  research  will  benefit  all  pertinent  areas  of 
study.   The  research  question  is  further  developed  Chapter 

III,  which  describes  the  experimental  methodology,  and  Chapter 

IV,  which  discussed  the  same  research  but  in  relation  to  the 
outcome  of  the  experiment . 
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III.    METHOD 

A.  Scope 

This  chapter  describes  the  experimental  design  used  in  the 
research.  The  experiment  was  divided  into  three  phases: 
training,  experiment,  and  debriefing.  The  rationale  for  the 
training  is  explained  in  the  previous  chapter.  Its  results 
will  be  described  in  this  chapter  in  section  G.  Section  I 
discusses  the  debriefing  phase,  and  section  J  is  a  summary  of 
this  chapter. 

B .  Hypotheses 

The  first  hypothesis  is  discussed  in  terms  of  the  lens 
model  index  for  achievement .  All  hypotheses  were  formulated 
on  the  individual  level . 

Hx:  Subjects  receiving  CFB  will  attain  higher  decision 
quality  than  those  receiving  only  OFB . 

H2 :  Subjects  will  perform  better  in  task  environments  of 
high  predictability  than  those  in  environments  of  low 
predictability . 

1 .   Description  of  Hypotheses 

Based  on  arguments  discussed  previously  in  Chapter  II, 
we  postulate  that  performance  will  improve  in  the  cognitive 
feedback  condition.   Performance  in  this  sense  means  that  the 
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subject's  responses  will  more  closely  resemble  the  committee 
score.  Similarly,  the  subject's  policy  will  resemble  the 
committee  policy. 

Section  C.l  of  Chapter  II  reviews  Streufert's  findings 
that  the  presence  of  irrelevant  information  is  more 
detrimental  to  decision  makers  in  environments  of  higher  task 
difficulty.  It  is  proposed  in  H2  that  this  effect  will  also 
hold  true  to  differences  in  task  predictability. 

C.   Experimental  design 

The  purpose  of  this  section  is  to  describe  the 
experimental  design  of  the  experiment.  The  experiment  has 
three  phases:  training  phase,  experimental  phase,  and 
debriefing. 

1 .   Description  of  design 

The  experimental  design  has  two  components  -  between 
subjects  effects  and  within  subjects  effects.  The  design 
chosen  to  accomplish  this  was  a  2  (OFB  vs  CFB)  X  2  (High  vs 
Low  Predictability)  X  3  (Block  sequence)  factorial  design. 

The  between-sub jects  effect  is  the  difference  in 
performance  between  those  subjects  given  cognitive  feedback 
and  those  only  given  outcome  feedback.  Another  between- 
subject  effect  is  the  difference  in  performance  between  those 
subjects  in  the  high  predictability  task  environment  and  those 
in  the  low  predictability  environment.  The  between  subjects 
design  is  described  by  Figure  1  below. 
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CFB/LO 

OFB/HI 

OFB/LO 

Figure  1 

As  well  as  determining  if  systematic  differences  exist 
among  experimental  conditions,  this  study  also  seeks  to  study 
the  effect  within  each  condition  over  time.  This  within- 
sub  jects  design  involves  measurements  that  occur  over  time. 
Time  of  response  will  be  measured  for  all  subjects  to  see  if 
learning  was  occurring  with  each  subject . 

The  within-sub jects  effect  of  irrelevant  information 
was  operationalized  by  varying  which  cue  (of  three)  was 
irrelevant  for  each  block.  The  order  that  each  subject 
receives  the  blocks  of  trials  was  then  operationalized  using 
a  Latin  Squares  design  following  the  procedure  in  given  in 
Kirk  (1982) . 

The  following  figure  demonstrates  the  separation  of 
within-sub jects . 

Group  1   Group  2   Group  3 
Task  order:         Cue  2    Cue  3     Cue  1 

Cue  1    Cue  2     Cue  3 
Cue  3    Cue  1     Cue  2 
Therefore,  group  1  gets  the  order  2,  1,  3;  group  2  gets  the 
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order  3,  2,  1;  and  group  3  gets  the  order  1,  3,  2.  For 
example,  subjects  in  group  1  will  receive  a  task  sequence  of 
the  following  order:  in  block  1,  cue  2  was  irrelevant;  in 
block  2,  cue  1  was  irrelevant;  and  in  block  three,  cue  three 
was  irrelevant . 

2 .   Experimental  setting 

The  sessions  were  conducted  in  a  closed  room  in  the 
presence  of  the  experimenter.  The  training  and  experimental 
phases  were  conducted  on  an  IBM  compatible  personal  computer. 
Brief  instructions  were  provided  by  the  experimenter  at  the 
beginning  of  the  training  phase.  From  that  point,  the  subject 
used  the  instructions  provided  and  on-screen  help  provided  by 
the  software  to  progress  through  the  experiment .  The  software 
was  written  by  the  experimenters .  The  debriefing  phase 
involved  filling  out  a  brief  questionnaire  attached  to  the 
instruction  booklet. 

D .   Subjects 

1 .   Choice  of  subjects 

The  experiment  was  conducted  with  sixty  subjects  at 
the  Naval  Postgraduate  School.  Each  subject  participated 
individually.  Each  participant  was  assigned  to  one  of  four 
cells  of  subjects.  These  four  blocks  are  described  in  Figure 
1 .  Subjects  were  assigned  to  one  of  four  between-sub jects 
conditions  and  one  of  three  within-sub jects  conditions. 
Randomization   was   ensured   in  both   of  these   assignment 
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procedures . 

Information  of  demographics  and  task  specific  factors 
that  may  impinge  on  the  results  of  the  experiment  was 
collected.  Demographic  factors  studied  for  interaction  in  the 
experiment  include:  age,  sex,  full-time  work  experience, 
familiarity  with  computers,  and  time  lapse  since  undergraduate 
study.  The  task-specific  factor  was  related  to  whether  the 
subject  had  any  previous  experience  in  the  task. 
2 .   The  use  of  students  as  subjects 

One  limitation  of  the  study  centers  around  the  use  of 
students  in  a  laboratory  environment.  All  participants  had  a 
minimum  of  four  years  management  experiment.  The  issue  is  the 
extent  to  which  it  is  possible  to  make  a  reasonable  comparison 
between  the  subjects  of  the  experiment  and  real-life  decision 
makers . 

Previous  research  using  graduate  students  as 
surrogates  for  managers  failed  to  find  any  significant 
difference  between  the  two  groups  in  making  production 
scheduling  decisions .  It  is  reasonable  to  assume  that 
graduate  students  are  acceptable  as  representative  of  decision 
makers . 

It  is  difficult  to  claim  external  validity  for 
laboratory  studies.  The  bulk  of  the  research  conducted  in 
support  of  the  premises  which  frame  the  research  question  is 
conducted  in  a  laboratory  setting.   The  similarities  between 
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other  research  in  the  area  of  irrelevant  research  and  this 
research  indicates  that  it  is  no  less  suitable  for 
generalization  to  real-life. 

E .   Task 

The  following  section  describes  in  detail  the  experimental 
task  (see  Appendix) .  Also  described  are  the  different  task 
environments  in  which   subjects  groups  performed. 

1 .   Description 

The  instruction  booklet  given  to  each  subject  varied 
based  on  the  task.  Subjects  allowed  to  use  cognitive  feedback 
were  given  additional  instructions  describing  the  different 
types  of  feedback  at  their  disposal  and  the  function  keys  to 
access  them.  The  debriefing  questionnaire  was  also  different 
for  the  two  different  types  of  subjects.  The  only  difference 
in  the  debriefing  questions  pertained  to  how  useful  the 
subject  found  the  types  of  feedback  were  in  making  their 
decision  rules. 

The  task  presented  to  the  subject  was  a  candidate 
screening  scenario.  The  subject  was  given  a  set  of  three  cues 
representing  three  scores  of  a  potential  job  applicant.  The 
three  scores  ranged  from  1  to  9 .  Each  of  the  three  scores 
represented  an  applicants  score  or  rating  on  each  of  three 
variables  or  conditions:  experience  relevant  to  the  position 
(variable  1) ,  general  abilities  test  score  (variable  2) ,  and 
interviewer  rating  of  management  abilities   (variable  3)  . 
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Based  on  these  ratings  the  subject  was  then  required  to  award 
an  overall  rating  of  these  candidates  on  a  scale  of  1  to  9 . 

In  the  training  phase  these  sets  of  three  cues  were 
presented  to  the  subject  one  set  at  a  time.  The  subject  gave 
his  response  and  the  committee  score  was  then  shown.  In  the 
experimental  phase,  the  sets  of  cues  were  blocked  into  groups 
of  26  applicants.  This  allowed  for  the  subject  to  see  the 
effects  of  his  policy  over  several  cases .  There  was  no 
irrelevant  cue  in  the  training  phase. 

F .   Feedback 

The  use  of  cognitive  feedback  was  provided  to  half  of  the 
subjects  in  the  experiment.  Feedback  was  explained  to  the 
subject  and  made  available.  It  was  given  to  the  subject  only 
on  request,  and  was  available  in  a  variety  of  forms.  The 
various  types  of  feedback  are  described  in  the  next  sections. 

1 .   Subject  policy  feedback 

The  purpose  of  policy  feedback  is  to  allow  the  subject 
to  see  the  policy  he  is  using  in  order  to  refine  and  improve 
upon  it.  This  type  of  feedback  was  presented  to  the  subject 
in  the  form  of  a  bar  graph.  Each  cue  was  represented  on  the 
bar  graph  in  a  different  color  or  shade  (see  Figures  2  and  3 
below) .  The  size  of  each  shade  was  determined  by  the  weight 
that  the  subject  was  placing  on  this  variable  in  arriving  at 
his  overall  score  for  the  applicant.  If  the  cue  has  negative 
value,  i.e.  a  high  score  in  a  variable  would  lower  the  overall 
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Figure   2 

score,  it  was  shown  in  appropriate  size  for  its  negative 
weight,  but  below  the  positive  values. 

In  previous  research  conducted  on  representation  of 
cognitive  feedback,  it  was  found  that  visual  information  was 
the  most  useful  to  the  subject  (Doherty  and  Balzer,  1988) . 
Hammond  (1971)  stressed  the  importance  of  giving  the  subject 
a  picture  of  his  cognitive  processes.  For  this  reason,  the 
graphical  representation  shown  in  the  above  figure  was  used. 
Each  of  the  three  cues  was  clearly  distinguishable,  making  the 
figure  clear  and  easy  to  understand. 

Cue  weights  were  calculated  as  follows: 

(a)  Beta  weights  were  calculated  from  a  multiple 
regression  of  cue  values  and  the  subject's  estimates. 

(b)  The  weights  were  then  transformed  to  represent 
percentages  of  the  sum  of  the  squared  weights.   Thus,  beta 
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values  of  0.4,  0.8,  and  0.2  were  shown  as  0.2,  0.75,  and  0.05. 

(c)  Transformed  weights  on  the  three  cues  were  then 
displayed  as  a  horizontal  stacked  bar,  on  a  1-100  scale  (i.e., 
the  transformed  weights  added  up  to  100) . 

This  form  of  feedback  is  described  as  a  type  of 
Cognitive  Information  (CI)  in  Chapter  II.  This  type  of 
feedback  provides  the  subject  with  the  function  forms  that 
relate  his/her  judgment  to  the  cues.  The  other  type  of 
cognitive  information  discussed  in  Chapter  II  is  consistency 
information . 

2 .   Consistency  feedback 

Consistency  feedback  gives  the  subject  an  indication 
as  to  how  well  he  was  conforming  to  his  decision  policy.  It 
may  not  be  inherently  obvious  that  different  sets  of  responses 
can  generate  the  same  policy,  but  one  set  of  responses  may  be 
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much  more  tightly  grouped  around  the  committee  scores.  Thus, 
the  consistency  feedback  will  provide  the  subject  an 
indication  as  to  how  consistent  he  was  with  his  mental  model. 
It  was  designed  also  to  give  some  assistance  in  improving  his 
consistency.  Both  of  these  goals  were  achieved  by  providing 
the  subject  with  consistency  scores  on  individual  cases.  In 
other  words,  the  subject  will  be  given  the  values  that  he 
would  have  generated  if  he  conformed  strictly  to  his  decision 
rule . 

Consistency  feedback  is  a  form  of  CFB  called  cognitive 
information  (CI)  by  Doherty  and  Balzer  (see  section  2.4.)  In 
terms  of  the  lens  model,  consistency  information  is  R,  or  R2. . 
It  can  be  expressed  in  terms  of  the  correlation  indices  of 
predictability  of  the  subject.  The  use  of  consistency 
"scores"  was  chosen  to  make  it  easy  for  the  subject  to 
understand  and  use. 

Consistency  scores  were  calculated  as  follows: 

(a)  First  beta  weights  were  derived  from  a  multiple 
regression  of  cue  values  and  the  subject's  estimates. 

(b)  Criterion  values  were  computed  by  multiplying  cue 
values  with  respective  beta  weights. 

3 .   Committee  decision  policy 

The  purpose  of  this  type  of  feedback  was  to  provide 
the  subject  with  a  reference  to  model  his  policy  after.  It 
was  stated  in  the  instructions  that  the  purpose  is  to  model 
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the  decisions  of  the  committee .  This  type  of  feedback  was 
presented  to  the  subject  in  the  same  manner  that  his  own 
policy  was  presented,  i.e.  through  the  use  of  a  bar  graph.  As 
before,  each  cue  was  given  a  section  of  the  bar  graph 
proportional  to  the  weight  that  the  committee  places  on  that 
cue.  In  every  set  of  cues,  two  of  the  three  were  weighted 
equally  and  the  third  was  irrelevant. 

This  type  of  feedback  is  called  task  information  (TI) 
by  Doherty  and  Balzer  (see  section  2.4) .   It  is  accepted  as 
the  most  useful  in  effecting  a  change  in  behavior.  This 
information  provides  the  subject  the  function  forms  relating 
the  criterion  to  the  cues . 

4 .  Outcome  feedback 

The  purpose  of  this  type  of  feedback  was  to  provide 
the  subject  some  indication  as  to  his  performance,  also  known 
as  the  committee  score.  This  form  of  feedback  was  presented 
to  the  subject  as  a  single  number  value.  This  value  was 
calculated  by  summing  the  result  of  multiplying  each  cue  by 
its  beta  weight.  This  type  of  feedback  gives  the  subject  an 
indication  of  his  achievement.  This  value  is  denoted  as  R^  in 
the  lens  model . 

5 .  Combined  feedback 

The  purpose  of  this  feedback  was  to  allow  the  subject 
to  view  two  different  types  of  feedback  at  one  time.  For 
example,  the  subject  can  ask  to  see  the  committee  policy  and 
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his  own  policy  side  by  side.  This  was  a  convenience  provided 
the  subject  to  allow  him  the  best  conditions  for  refining  his 
policy.  The  types  of  combined  feedback  allowed  to  the  subject 
are:  committee  and  subject  policy  rules,  information  on 
subject  decision  rule  and  consistency. 
€ .   Multiple  requests  for  feedback 

The  subject  was  allowed  to  request  feedback  many 
times .  The  only  restraint  on  this  was  that  once  a  subject  has 
requested  outcome  feedback,  he  may  no  longer  make  revisions  to 
those  scores  for  which  the  committee  score  was  given. 

G .   Training 

1 .   Basis  for  training 

The  purpose  of  training  in  the  experiment  was  ensure 
that  all  of  the  subjects  begin  the  experiment  with  the  same 
knowledge  of  the  task.  Due  to  the  context  of  the  experiment, 
some  subjects  may  come  into  the  experiment  with  some 
preconceived  ideas  about  which  variables  should  have  greater 
weights  in  their  policy,  i.e.  some  may  feel  that  the  interview 
score  is  more  important  than  job  experience.  In  the  training 
phase,  all  cues  were  weighted  equally  (there  was  no 
irrelevant  cue)  .  At  the  end  of  the  training  phase,  all 
subjects  should  have  adequate  knowledge  of  the  task  to  perform 
well . 
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2 .   Training  procedure 

During  the  training  phase,  all  subjects  received  the 
same  set  of  data.  Each  subject  was  asked  to  respond  to  70 
cases .  The  subject  was  required  to  give  each  candidate  an 
overall  score  from  1  to  9 .  This  was  followed  by  the  actual 
committee  score.  The  cues,  subject  response,  and  committee 
score  would  remain  on  the  screen  until  the  subject  presses  a 
key  to  bring  on  the  next  set.  Thus,  the  subject  could  take 
as  long  as  he  needed  for  the  training  but  could  only  view  one 
case  at  a  time. 

H.   Experimental  phase 

1 .   Experimental  procedure  common  to  all  subjects 

The  experiment  was  conducted  over  three  blocks  of 
twenty-six  cases  each.  The  order  in  which  these  blocks  were 
presented  to  the  subject  was  randomized  in  order  to  discount 
the  effect  of  order  effects.  Appendix  A  contains  details  of 
the  instructions  given  to  each  of  the  subjects  at  the 
beginning  and  end  of  the  experiment .  Each  block  required  the 
subject  to  perform  a  repetition  of  one  type  of  task.  Each 
subject  would  sit  at  the  terminal  and  read  the  introductory 
screen  explaining  how  to  progress  through  the  experiment. 
After  pressing  a  key,  the  subject  was  presented  with  twenty- 
six  sets  of  three  cues.  The  screen  was  split  down  the  middle 
so  that  thirteen  cases  appear  on  each  side.  Each  cue 
represents  a  value  as  described  in  section  C.l.   The  subject 
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was  allowed  to  make  an  overall  evaluation  of  the  candidate  and 
award  a  score  between  1  and  9 .   The  subject  may  use  the  arrow 
keys  to  move  from  one  case  to  another.   No  requirements  exist 
that  require  the  subject  to  evaluate  the  cases  in  order. 
2 .   Use  of  cognitive  feedback 

After  ten  scores  have  been  filled  in  by  the  subject, 
he  can  request  one  of  the  types  of  feedback  as  described  in 
section  3.5.  To  view  the  options  for  feedback,  the  subject 
need  only  press  the  END  key.  If  no  feedback  was  required,  the 
subject  may  return  to  the  experiment  by  pressing  the  N  key. 
The  subject  may  request  outcome  feedback  at  any  time,  but  may 
no  longer  change  his  scores  after  doing  so.  When  all  cases 
have  been  evaluated,  the  subject  again  presses  the  END  key  and 
then  was  led  through  a  series  of  questions  to  ensure  that  he 
was  through  with  the  block  and  then  the  next  block  was 
presented.  No  time  limit  was  placed  on  the  subject.  After 
completion  of  each  block,  the  subject  was  required  to  fill  out 
a  short  questionnaire. 

I .   Debriefing 

The  objective  of  the  debriefing  phase  was  to  have  the 
subject  provide  a  description  of  his  own  policy  and  thought 
processes  as  he  progressed  through  the  experiment. 
Information  provided  in  this  section  will  give  insight  into 
how  the  subjects  viewed  different  aspects  of  the  experiment . 
The   debriefing   results   will   provide   the   experimenter 
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information  concerning  how  clear  all  phases  of  the  experiment 
were  to  the  subjects . 

J .   Summary 

This  chapter  explained  the  experimental  environment.   Of 
special  interest  in  this  chapter  are  the  following  points: 

(1)  The  experiment  included  four  different  groups  of  fifteen 
subjects  each.  Within  each  of  these  groups,  the  subjects  were 
further  differentiated  by  the  order  in  which  the  blocks  were 
presented.  This  is  to  discount  order  effects,  should  one 
block  of  data  be  significantly  different  from  another. 

(2)  The  first  phase  of  the  experiment  was  the  training  phase. 
The  purpose  of  this  phase  is  to  ensure  that  all  subjects  enter 
the  experiment  with  equivalent  knowledge  of  the  task  and  no 
prejudices  about  the  simulated  task  were  carried  into  the 
experiment.   The  training  consisted  of  seventy-two  trials. 

(3)  Cognitive  feedback  was  provided  (for  half  the  subjects) 
in  three  different  forms:  subject  policy  information,  subject 
consistency  information,  and  committee  policy  information. 

(4)  Subjects  all  performed  the  same  task  in  the  simulation. 
They  evaluated  potential  job  candidates  based  on  three  values: 
work  experience,  test  scores,  and  interview  score.  Outcome 
feedback  was  provided  for  all  subjects  upon  request. 
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IV.    RESULTS 

A.   Statistical  Model 

The  following  analysis  of  variance  (ANOVA)  model  suited 
for  multiple  Latin  Squares  was  used  to  test  hypothesis: 

yljM1)m  =  \i  +  a±   +  fij  +  rk  +  xx  +  aB  +  (rx)^  + 
(ra)ki  +  (rii)kj  +  eijk(1)B   where: 
\l   is  constant, 

a±  is  the  sequence  of  experimental  tasks  (see  Chapter  III, 
p. 21), 

15 j  is  the  order  of  the  task  ( j  =  1,  .  .  .3)  , 
Tk  is  the  feedback  condition, 

xx  is  the  experimental  task  (i.e.,  cue  1  irrelevant,  cue  2 
irrelevant,  etc.), 

<jm  is  the  experimental  participant  (or  subject,  m  =  1,  .  .  .,53)  , 
and 
eijk(Dm  is  the  experimental  error  term. 

This  model  was  run  for  lens  model  indices  achievement 
(Ra)  ,  consistency  (R.)  ,  and  matching  (G)  ,  and  number  of 
iterations  (ITER)  performed  by  the  subject  within  a  block. 
The  analyses  were  conducted  on  SASR  statistical  software, 
using  the  General  Linear  Models  procedure.  The  general  linear 
models  procedure  was  chosen  because  the  number  of  subjects 
assigned  to  each  condition  varied.    The  assignments  of 
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subjects  to  blocks  was  as  follows:   CFB/H  =  12,  CFB/L  =  14, 
OFB/H  =  13,  OFB/L  =  12. 

B.   Results 

1 .  Training 

The  results  of  the  training  phase  are  summarized  in 
table  1 .  The  same  dependent  variables  were  analyzed  in  the 
training  phase  data.  Different  sets  of  cues  were  given  to  the 
subjects  in  the  training  phase  based  on  assignment  to  task 
predictability  conditions  (i.e.  all  subjects  in  the  high 
predictability  conditions  received  the  same  cues  in  the 
training  phase) .  All  subjects  received  one  block  of  decisions 
during  the  training  phase.  The  only  significant  effects  seen 
in  the  training  data  was  a  significant  variance  between  groups 
(high/low  predictability)  in  the  training  phase.  Analysis  of 
training  phase  data  is  displayed  in  Table  1 . 

2 .  Experimental  Results 

Tables   2-6   summarize   the   means   and   standard 
deviations  for  all  of  the  independent  variables  for  the 
experimental  phase.   Graphical  representation  of  this  data  is 
provided  for  each  set  of  data  in  the  tables . 
a.  Means  and  Standard  Deviations 
Achievement 

Achievement  is  the  measure  of  the  agreement  between 
the  criterion  values  and  responses  of  the  subject  over  n 
observations.  An  inspection  of  the  means  for  achievement  (RA) 
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Table  1 


TRAINING  DATA  ANALYSIS 


Class     Levels     Values 

GROUP  4     CFB/H  CFB/L  OFB/H  OFB/L 

SEQ  3     12  3 

Number  of  observations  In  data  set  -  53 


Dependent  Variable:  RA 


Sum  of 

Mean 

Source 

DF 

Squares 

Square 

F  Value 

Pr  >  F 

Model 

11 

0.490 

0.0446 

4.07 

0.0005 

Error 

41 

0.449 

0.0109 

Crctd  Total 

52 

0.939 

GROUP 

3 

0.4096 

0.1365 

12.48 

0.0001* 

SEQ 

2 

0.0123 

0.0062 

0.56 

0.5733 

GROUP* SEQ 

6 

0.0682 

0.0113 

1.04 

0.4141 

Dependent  Variable: 

G 

Model 

11 

0.010 

0.0009 

0.79 

0.6478 

Error 

41 

0.048 

0.0012 

Crctd  Total 

52 

0.058 

GROUP 

3 

0.005 

0.0017 

1.50 

0.2277 

SEQ 

2 

0.0017 

0.0008 

0.72 

0.4949 

GROUP*SEQ 

6 

0.0032 

0.0005 

0.46 

0.8345 

Dependent  Variable: 

RS 

Model 

11 

0.0918 

0.0080 

0.97 

0.4896 

Error 

41 

0.3537 

0.0090 

Crctd  Total 

52 

0.4454 

GROUP 

3 

0.0321 

0.0107 

1.24 

0.3064 

SEQ 

2 

0.0130 

0.0065 

0.76 

0.4763 

GROUP* SEQ 

6 

0.0466 

0.0078 

0.90 

0.5036 

-  Significant  at  0.05  level 


Table  2   MEANS  AND  STANDARD  DEVIATIONS  FOR  ACHIEVEMENT 


Group 


Block  1 


Block 


Block  3 


CFB/H 
CFB/L 
OFB/H 
OFB/L 


.841(.106) 
.576(.207) 
.768(.117) 
.502(.169) 


.818(.192) 
.607(.137) 
.744(.081) 
.470(.231) 


.864(.131) 
.  609(.149) 
.709(.105) 
.457(.183) 


*  Mean (St .Dev. ) 


in  Table  2  show  that  subjects  in  the  CFB/H  condition  had  the 
highest  degree  of  accuracy.   The  graphical  representation  of 
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this  data,  in  Figure  4,  shows  that  achievement  in  the 
cognitive  feedback  conditions  improved  in  consecutive  blocks 
while  achievement  declined  in  both  outcome  feedback 
conditions.  This  result  supports  Hx  as  discussed  in  Chapter 
III.  From  the  graph  and  the  table  it  is  also  clear  that  for 
subjects  within  the  same  feedback  condition,  those  in  the  high 
predictability  condition  had  higher  values  for  achievement 
than  those  in  the  low  predictability  condition.  This  supports 
H2  as  discussed  in  Chapter  III. 


Achievement 


Mean  Score 


0.9 


0.8 


0.7 


0.5tJ 


L 


0.4 

BLOCK   1 


BLOCK  2 


BLOCK  3 


CFB/H 


CFB/L 


OFB/H 


OFB/L 


Figure  4    Experimental  results  for  achievement 
Consistency 

Consistency  is  defined  as  the  multiple  correlation 
between  the  cues  and  the  criterion.  Consistency  for  a  subject 
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is  calculated  from  estimates  made  by  the  subject  in  his/her 
final  iteration  in  a  given  block.  The  values  for  consistency 
in  Table  3  indicate  that  subjects  in  the  CFB  conditions  were 
the  most  consistent  in  the  utilization  of  their  decision 
rules,  as  defined  by  the  multiple  regression  equation.  The 
graphical  representation  of  this  data,  in  Figure  5,  shows  that 
consistency  improved  in  subsequent  blocks  in  the  cognitive 
feedback  condition. 

Table  3      MEANS  AND  STANDARD  DEVIATIONS  FOR  CONSISTENCY 


Group                    Block  1  Block   2  Block  3 

CFB/H  .914 (.086)  .911(.077)  .944(.032) 

CFB/L  .914 (.046)  .885 (.107)  .923 (.053) 

OFB/H  .898 (.045)  .886  (.049)  .866  (.073) 

OFB/L  .867(.101)  .839(.105)  .840(.117) 


*  Mean (St .Dev. ) 

The  highest  mean  for  consistency  was  found  in  the 
CFB/H  Block  3  condition,  while  the  lowest  was  found  in  the 
OFB/L  Blocks  2  and  3  condition.  Subjects  in  the  CFB  condition 
tended  to  become  extremely  consistent  in  the  final  block,  with 
very  little  variation,  as  indicated  by  the  small  standard 
deviation . 

Matching 

Matching  is  defined  as  the  correlation  between  model 

estimates  of  the  subject  with  the  model  of  the  environment 

(committee  policy) .    The  values  for  matching  in  Table  4 
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Consistency 


Mean  Score 


0.95 
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BLOCK  1 


_i_ 


BLOCK  2 


BLOCK  3 


—  CFB/H 


CFB/L   -*-  OFB/H   -S~  OFB/L 


Figure  5  Experimental  results  for  consistency 
indicate  that  subjects  in  the  high  predictability  had  the 
greatest  success  in  matching  the  committee  policy  with  their 
own  policy.  Graphical  representation  of  this  data,  in  Figure 
6,  shows  that  matching  was  best  with  cognitive  feedback  and 
high  predictability  conditions.  The  higher  degree  of  matching 
between  subject  and  committee  in  the  cognitive  feedback 
condition  support  H1   as  suggested  in  Chapter  III. 

For  cognitive  feedback  -  low  predictability 
conditions,  matching  improved  with  each  subsequent  block.  It 
should  be  noted  that  although  the  matching  index  was  on  the 
average  higher  in  the  OFB/H  condition  than  in  the  CFB/L 
condition,  in  the  third  block  subjects  in  the  CFB/L  condition 
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Figure  6    Experimental  results  for  matching 

Table  4     MEANS  AND  STANDARD  DEVIATIONS  FOR  MATCHING 


Group 


Block  1 


Block   2 


Block  3 


CFB/H 
CFB/L 
OFB/H 
OFB/L 


.967(.040) 
.809(.201) 
.877(.148) 
.722(.181) 


*  _ 


Mean (St .Dev. ) 


.917 (.032) 
.832(.162) 
.878(.075) 
.692(.273) 


.948 (.120) 

.870(.157) 
.869(.lll) 
.793(.099) 


were  able  to  surpass  the  matching  ability  of  subjects  in  any 
of  the  OFB  conditions. 

Iteration 

Iteration  is  described  as  the  number  of  times  that  a 
subject  requests  feedback  in  a  given  block.  The  data  in  table 
V  indicates  that  subjects  in  the  cognitive  feedback  condition 
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TERATIONS 


Mean  Score 


BLOCK   1 


CFB/H 


BLOCK  2 


CFB/L   -*-  OFB/H    -B~  OFB/L 


BLOCK  3 


Figure  7    Experimental  results  for  iteration 

completed  each  block  in  the  fewest  number  of  iterations.  This 
observation  does  not  indicate  that  the  subjects  in  the 
cognitive  feedback  conditions  completed  the  blocks  in  the 
shortest  amount  of  time.  The  graphical  representation  of  this 
data  is  in  Figure  7. 

Table  5      MEANS  AND  STANDARD  DEVIATIONS  FOR  ITERATION 


Group 


Block  1 


Block   2 


Block  3 


CFB/H 
CFB/L 
OFB/H 
OFB/L 


11.9(7.90) 
9.29(7.18) 
8.23(7.63) 
13.29.86) 


8.21(4.92) 
6.86(4.00) 
15.9(8.97) 
18.3(10.4) 


7.93(3.71) 
7.79(5.54) 
14.5(7.37) 
13.5(9.28) 


Mean (St .Dev. ) 


41 


DIFFERENCE 
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Figure  8    Experimental  results  for  difference 

Difference 

Difference  is  described  as  the  difference  between  the 
weights  used  by  the  subject  and  the  committee  weights.  Thus, 
a  small  value  for  DIFF  would  indicate  that  the  subject 
utilized  similar  cue  weights  as  the  committee.  This  value  is 
smallest  for  subjects  in  the  cognitive  feedback  condition. 
Graphical  representation  of  this  data  is  found  in  Figure  5. 

The  means  for  achievement  indicate  that  the  main 
effects  are  due  to  groups  (task  condition)  and  subject 
(variations  from  one  subject  to  another) .  The  same  is  true 
for  consistency  and  matching. 
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Table    6 

MEANS    AND    STANDARD 

DEVIAITONS    FOR   DIFFERENCE 

Group 

Block   1 

Block      2                     Block   3 

CFB/H  .497(.193)  .563(.287)  .524(.250) 

CFB/L  .666(.292)  .613(.314)  .531(.263) 

OFB/H  .767(.311)  .758(.209)  .785(.278) 

OFB/L  .834(.277)  .863(.413)  1.14(1.29) 


*  -  Mean (St .Dev. ) 

Jb.  ANOVA  Results 

The  ANOVA  was  conducted  on  the  same  variables  in 
the  previous  section,  namely:   RA,  RS,  ITER,  G,  and  DIFF .   The 
results  of  the  ANOVA  are  summarized  in  tables  7-9. 
Table  7      ANOVA  RESULTS  FOR  ACHIEVEMENT 


Sum  of 

Source    of   Variation 

DF 

Squares 

F   Value 

Pr    >   F 

R-Square 

Model 

62 

4.75825204 

3.95 

0.0001* 

0.718280 

GROUP 

3 

2.99613739 

51.37 

0.0001* 

BLOCK 

2 

0.00370189 

0.10 

0.9093 

SEQ 

2 

0.08812805 

2.27 

0.1092 

TASK 

2 

0.03695659 

0.95 

0.3901 

LNAME 

47 

1.57564399 

1.72 

0.0125* 

GROUP* BLOCK 

6 

0.05768413 

0.49 

0.8110 

*    -   Significant    0.05 

level 

The  ANOVA  for  accuracy  (RA)  showed  significant  main 
effects  for  group  (p, 0.0001)  and  participant  (p<0.05). 
Variances  in  accuracy  not  found  to  be  significant  were  between 
blocks,  sequence,  task,  and  group-block. 

ANOVA  for  matching  (G)  showed  that  the  variance  of 
matching  was  significant  in  the  same  categories  as 
consistency,  groups  (0.0001)  and  last  name  (0.0196).  Matching 
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did  not  vary  significantly  in  any  other  conditions  tested. 
Table  8     ANOVA  RESULTS  FOR  ACHIEVEMENT 


Sum  of 

Source  of  Variation 

DF 

Squares 

F  Value 

Pr  >  F 

R-Square 

Model 

62 

0.66091699 

2.84 

0.0001* 

0.647277 

GROUP 

3 

0.11970669 

10.64 

0.0001* 

BLOCK 

2 

0.00897107 

1.20 

0.3070 

SEQ 

2 

0.00463097 

0.62 

0.5416 

TASK 

2 

0.03546305 

4.73 

0.0110* 

LNAME 

47 

0.46653530 

2.65 

0.0001* 

GROUP *BLOCK 

6 

0.02560992 

1.14 

0.3465 

*  -  Significant  at  0 

.05  level 

ANOVA  for  consistency  showed  that  variance  was 
significant  between  groups  (0.0001),  task  (0.011),  and  last 
name  (0.0001)  Consistency  did  not  vary  significantly  between 
blocks,  sequence,  or  group-block. 

Table  9      ANOVA  RESULTS  FOR  MATCHING 


Sum  of 

Source  of  Variation 

DF 

Squares 

F  Value 

Pr  >  F 

R-Square 

Model 

62 

2.68665623 

2.20 

0.0002* 

0.587311 

GROUP 

3 

0.87393415 

14.81 

0.0001* 

BLOCK 

2 

0.04132201 

1.05 

0.3537 

SEQ 

2 

0.09566424 

2.43 

0.0932 

TASK 

2 

0.08119316 

2.06 

0.1325 

LNAME 

47 

1.52604186 

1.65 

0.0196* 

GROUP* BLOCK 

6 

0.06850081 

0.58 

0.7450 

*  -  Significant  at  0 

,05  level 

c .  Scheffe ' s   Tost 

Scheffe's  Test  is  a  posterior  test  performed  to 
determine  if  differences  within  a  variable  are  significant 
enough  to  account  for  observed  differences  in  experimental 
results.    Scheffe's  test  was  performed  between  the  following 
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variables  and  all  possible  combination  of  task  conditions: 
RA,  G,  RS,  ITER. 

In  achievement,  Scheffe's  Test  indicated  that  the 
means  were  significantly  different  from  each  other  (p  <  0.05) 
between  all  task  conditions.  In  matching,  Scheffe's  Test 
indicated  that  means  were  significantly  different  (p  <  0.05) 
in  all  conditions  except  when  compared  between  CFB/H  -  OFB/H 
and  CFB  -  OFB/H.  In  consistency,  all  means  were  significantly 
different  at  the  same  level  except  when  compared  between  CFB/H 
-  CFB/L,  CFB/L  -  OFB/H,  and  OFB/H  -  OFB/L. 
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V.    CONCLUSIONS 


Scope 


The  purpose  of  this  chapter  is  to  revisit  the  research 
topics  discussed  in  earlier  chapters  in  light  of  the  results 
of  this  research.  The  limitations  of  this  research  and  the 
potential  impact  on  potential  research  will  be  included  in 
this  chapter.  This  chapter  also  discusses  the  conclusions  of 
the  research  in  terms  of  the  majors  areas  of  study  with  which 
it  deals:  information  relevance  and  cognitive  feedback;  and 
limitations  of  this  research.  The  chapter  ends  with  a 
discussion  of  the  relevance  of  this  thesis  to  research  in 
computer  systems . 

B.   Conclusions 

1 .   Previous  Research 

We  argued  previously  that  while  research  in 
information  relevance  was  abundant,  research  in  finding  means 
of  reducing  the  effects  of  irrelevant  information  is  an 
important  next  step.  In  addition  to  this  research,  an  example 
of  such  an  effort  is  the  research  done  by  Gaeth  and  Shanteau 
(1984)  . 

The  results  of  the  research  performed  by  Gaeth  and 
Shanteau  is  particularly  of  importance  in  terms  of  improving 
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decision  quality  in  environments  where  irrelevant  information 
is  present.  In  their  research,  training  was  found  to  be  an 
effective  measure  in  improving  experienced  decision  makers 
decision  quality,  by  reducing  the  effects  of  irrelevant 
information.  Similarly,  in  this  research,  the  use  of 
cognitive  feedback  was  shown  to  be  effective  as  well  in 
improving  decision  quality  when  irrelevant  information  was 
present . 

2 .   Research  Question 

The  question  posed  by  this  study  was:  given  that 
irrelevant  information  serves  as  a  detriment  to  decision 
quality,  can  the  use  of  cognitive  feedback  reduce  the  adverse 
effects  of  this  detriment . 

C.   Findings  of  the  study 

Lens  Model    Indices 

Decision  quality  was  operationalized  in  terms  of  the 
following  three  lens  model  indices:  achievement,  consistency, 
and  matching. 

•  Achievement:  Achievement  was  shown  to  be  the  highest  in 
conditions  where  subjects  in  the  high  predictability  task 
environment  received  cognitive  feedback.  Achievement 
improved  in  subsequent  blocks  in  the  cognitive  feedback 
condition  while  declining  in  the  outcome  feedback 
condition . 

•  Matching:  Subjects  receiving  cognitive  feedback  recorded 
higher  matching  index  than  those  receiving  only  outcome 
feedback. 

•  Consistency:  Consistency  was  highest  for  subjects  in  the 
cognitive  feedback  conditions.   Further,  for  subjects  in 
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the  cognitive  feedback  condition  consistency  improved  in 
subsequent  blocks. 


D .   Application  of  Cognitive  Feedback 

Brehmer  has  been  a  major  contributor  to  research 
concerning  cognitive  feedback.  His  work  emphasizes  the 
importance  of  cognitive  feedback  to  the  point  of  dismissing 
outcome  feedback  entirely.  Only  cognitive  feedback  can  give 
a  person  insight  into  relations  between  variables.  Further, 
only  cognitive  feedback  is  of  use  in  describing  subject 
consistency . 

In  support  of  Brehmer,  Doherty  and  Balzer  state  that  the 
human  learning  process  is  imperfect  -  "we  need  help"  (Doherty 
and  Balzer,  1988)  Their  research  also  points  out  the  strong 
link  between  consistency  and  cognitive  feedback. 

Cognitive  feedback  can  be  operationalized  in  three  forms 
(discussed  in  chapter  two)  :  task  information,  cognitive 
information,  and  functional  validity  information.  Various 
research  on  cognitive  feedback  has  stressed  each  of  these  as 
important  in  different  instances.  While  there  has  been  some 
agreement  in  the  research,  it  is  clear  that  future  research 
should  focus  on  the  distinctions  between  these  forms  of 
cognitive  feedback  and  the  importance  of  each. 

The  use  of  technology  in  the  application  of  cognitive 
feedback  is  an  exciting  enterprise  with  potential  for  enormous 
growth.   Personal  decision  support  systems  designed  to  assist 
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the  user  in  decisions  such  as  major  purchases  or  solving 
interpersonal  problems  are  within  the  realm  of  possibility  in 
the  near  future.  Providing  the  user  with  insight  into  his  or 
her  decision  processes,  with  the  promise  of  improving  decision 
quality,  is  appealing  to  designers  in  a  variety  of  areas. 

E.   Limitations  and  Recommendations  for  Further  Research 

It  is  in  the  best  interest  of  the  experimenter  to  control 
and  limit  the  scope  an  experiment  in  order  to  reduce  the 
chance  of  numerous  interaction  effects  or  effects  which  cannot 
be  explained  by  manipulation  of  the  dependent  variables.  The 
loss  associated  with  the  gains  of  control  of  scope  is  that 
only  a  small  part  of  a  large  picture  may  be  examined  at  one 
time.  The  issues  addressed  in  this  section  are  the  rest  of 
the  picture  -  those  issues  which  bear  some  significance  to  the 
research  topics  but  extend  beyond  the  scope  of  this  research 
1 .   Information  Relevance 

From  previous  research  on  information  relevance  is 
clear  as  to  the  adverse  effects  associated  with  irrelevant 
information.  Other  aspects  of  information  relevance  are  less 
clear  from  the  research  completed  to  date.  Further  research 
should  direct  emphasis  upon  the  following  points: 

•  Given  that  subjects  in  more  predictable  task  environments 
are  less  affected  by  irrelevant  information,  where  and  why 
is  this  distinction  seen?  Why  is  cognitive  feedback  more 
effective  in  more  predictable  environments. 

•  How  does  an  individual  determine  which  information  is 
relevant  and  which  is  not?    Do  methods  for  sorting 
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information  based  on  relevance  increase  in  complexity  as 
the  information  itself  increases  in  complexity? 

•  Overcoming  the  adverse  effects  of  irrelevant  information 
in  decision  making  tasks. 

•  The  impact  of  training  in  improving  task  performance  in 
task  environments  where  irrelevant  information  is  present. 

2 .   Cognitive  Feedback 

One  limitation  to  experimental  research  on  cognitions 
and  policy  formation  is  that  an  experimental  setting  leads 
subjects  to  form  policies  and  cognitions  differently  than  in 
the  real  world.  One  example  of  such  is  knowledge  acquisition. 
In  this  research,  for  example,  the  subject  is  given  three  cues 
from  which  he  must  make  a  decision.  The  subject  has  a  priori 
knowledge  that  he  will  be  given  sets  of  cues.  He  is  not 
expected  to  do  anything  to  get  this  information.  This  aspect 
of  the  experimental  setting  is  different  to  the  way  in  which 
individuals  gain  knowledge  in  the  real  world.  In  other  words, 
an  experimental  subject  may  be  led  to  know  that  he  or  she  is 
expected  to  form  a  policy,  and  may  even  be  led  as  to  what  form 
of  policy  is  expected. 

Another  limitation  of  the  study  of  human  cognitive 
processes  stems  from  the  use  of  the  Lens  Model .  The  Lens 
Model,  while  an  accepted  model  for  studying  human  behavior,  is 
rigid  and  limits  the  results  to  the  confines  of  a  finite 
number  of  coefficients. 

Cognitive  feedback  studies  thus  far  (including  this 
research)  have  not  explored  how  decision  makers  use  cognitive 
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feedback  (perhaps  because  of  the  difficulty  foreseen  in 
operationalization)  .  For  example,  when  and  why  does  a 
decision  maker  request  feedback?  Is  feedback  used  to  confirm 
an  idea  more  effective  than  feedback  used  initially?  What 
type  of  decision  makers  benefit  the  most  from  the  different 
types  of  feedback? 

An  area  for  further  research  is  the  use  of  cognitve 
feedback  in  conjunction  with  training  to  assist  people  to 
detect  irrelevant  information.  The  results  of  the  work  done 
with  training  discussed  in  this  thesis  indicates  that  there  is 
great  promise  in  this  venture. 

F .   Relevance  to  Research  in  Computer  Systems 

One  main  thrust  of  research  in  computer  systems  is  in 
decision  support  systems.  The  technology  in  decision  support 
systems  has  reached  a  point  where  the  competition  is  no  longer 
between  the  capabilities  of  the  hardware  or  the  power  of  the 
coding.  The  limiting  factor  in  most  interactive  software  is 
the  user.  Thus,  the  area  of  the  most  potential  for  growth  or 
additional  understanding  is  in  putting  more  power  in  the  hands 
of  the  user. 

Conversely,  power  can  be  taken  from  the  hands  of  the  user 
of  a  decision  support  system  through  the  presence  of 
irrelevant  information  (supported  by  this  research) .  The  use 
of  cognitive  feedback  shows  great  promise  in  the  field  of 
decision  support  systems .   Fundamental  to  this  is  a  greater 
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understanding  of  the  human  cognitive  process . 

6 .   Summary 

The  intent  of  this  research  has  been  to  study  decision 
making  across  different  task  situations.  Through  the 
employment  of  cognitive  feedback,  this  study  has  shown  that 
human  decision  making  processes  may  be  enhanced  by  simple 
decision  aids.  This  result  contributes  to  the  knowledge  base 
which  deals  with  cognitive  processes  and  decision  making.  A 
great  deal  is  still  unknown  about  the  way  humans  make 
decisions.  However,  from  this  research  and  others,  advances 
can  be  made  in  design  technology  for  computer  based  decision 
tools . 
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APPENDIX 

The  following  appendix  is  the  instruction  set  given  to 
subject  to  guide  them  in  completion  of  the  experiment.   The 
instruction  set  contained  additional  information  for  subjects 
in  the  cognitive  feedback  conditions.   This  additional 
information  will  be  in  bold. 
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YOUR  NAME  : 
SMC  NO. 


DESCRIPTION  OF  THE  SIMULATION 


*     This  simulation  seeks  to  gain  an  understanding  of  how 
decision  makers  use  information  for  making  decisions.   To 
investigate  the  use  of  information,  we  are  asking  a  number  of 
subjects  to  participate  in  a  variety  of  business  simulations. 
The  task  in  this  simulation  involves  screening  applicants  for 
a  set  of  positions  in  a  large  software  company.   You  are 
required  to  make  decisions  based  on  data  presented  to  you. 
The  simulation  has  three  parts. 

Part  one   is  the  training  session.   The  objective  here  is  to 
give  you  some  practice  on  how  to  make  decisions  in  this  task. 

Part  two  is  the  main,  or,  experimental  phase.   In  this  phase, 
you  make  decisions  based  on  an  actual  set  of  cases. 

Part  three   is  the  debriefing  phase. 


*    You  will  be  given  a  set  of  instructions  at  the  beginning 
of  each  part . 


*    A  brief  questionnaire  is  to  be  filled  out  at  the  end  of 
each  part . 


*     The  simulation  will  be  run  on  a  computer  terminal.   At 
all  times,  the  bottom  row  of  the  screen  will  prompt  you  about 
what  to  do  next . 


*  Please  follow  the  guidelines  strictly.   The  system 
prompts,  along  with  instructions  in  this  booklet,  will  guide 
you  at  every  stage . 

*  When  in  doubt,  ask  the  experimenter. 

***  please  turn  over  for  further  instructions  *** 
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PART  I:   TRAINING 


INSTRUCTIONS 


*    A  large  software  company  periodically  recruits  personnel 
for  a  variety  of  positions.   The  hiring  decisions  are  made 
from  a  large  pool  of  applicants.   Because  the  number  of 
applicants  is  large,  the  task  of  hiring  is  split  into  two 
stages : 

First,  a  committee  made  up  of  experts  from  the  human  resource 
division  screens  the  candidates . 

The  results  of  the  screening  exercise  are  sent  to  individual 
departments.   These  departments  then  conduct  further 
interviews  and  make  the  final  hiring  decisions. 


*  The  simulation  focuses  on  the  first  part  of  the  hiring 
task,  i.e.,  screening  candidates.  In  the  training  phase  of 
the  simulation,  the  committee  of  experts  would  like  to  train 
you,  so  that  you  can  perform  the  screening  task  in  the  same 
way  as  they  do.  The  objective  of  the  training  is  to  ensure 
that  your  decisions  match  decisions  they  would  have  made. 


*     The  committee  of  experts  uses  the  following  three 
variables  in  screening  a  particular  candidate: 


Variable  1 
Variable  2 
Variable  3 


Experience  relevant  to  the  position. 

General  abilities  test  score. 

Interviewer  rating  of  management  abilities 


Decisions  you  make  must  also  be  based  on  these  three  variables 
only . 

*     Each  variable  is  presented  to  the  committee  (and 
therefore,  to  you)  as  a  range  of  values  between  1-9,  where  the 
values  imply  the  following: 

1   ■ ■  9 

Very  Very 

Poor  Good 
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*     You  will  be  given  profiles  of  several   applicants.   The 
profile  of  each  applicant  will  be  presented  as  a  set  of  values 
on  the  three  variables  described  above.   This  simulation 
requires  you  to  rate  the  candidate  on  an  overall  basis,  on  a 
scale  of  1-9 ,  wherein: 


1 
Very 
Poor 


■  9 
Very 
Good 


For  Example, 


Variable 

1 

Variable 

2 

Variable 

3 

Dec.  Maker's 

(given) 

(given) 

(given) 

Score 

9 

6 

8 

7 

4 

8 

9 

6 

. 

. 

. 

? 

• 

• 

• 

etc . 

For  the  first  candidate,  Variable  1  had  a  9,  Variable  2  had  a 
6,  and  Variable  3  an  8.   The  decision  maker  gave  an  overall 
score  of  7  for  the  candidate.   And  so  on.   (By  the  way,  the 
scores  given  above  are  just  random,  and  not  examples  of  actual 
values  the  committee  would  have  given) . 


*     In  the  training  phase,  you  will  be  given  several  cases 
(i.e.,  candidates),  one  at  a  time.  After  you  enter  a  score  on 
each  case,  the  system  will  provide  you  with  the  actual  value 
given  by  the  committee  (on  the  right  hand  side  of  your  score)  . 

Your  task  is  to  predict  the  committee  values  as  closely  as 
possible. 


*     Your  first  predictions  will  probably  be  guesses. 
However,  as  you  work  through  the  cases,  you  will  be  able  to 
learn  the  relationships  between  the  variables  and  the  actual 
committee  scores .  You  should  try  to  come  as  close  as  possible 
to  the  actual  committee  scores .  The  task  has  been  arranged  so 
that  you  can  learn  to  make  predictions  with  moderate  accuracy. 

As  in  the  real  world,  though,  predictions  (of  committee 
judgments)  cannot  be  perfectly  accurate  all  the  time. 


*    After  completing  the  task,  please  answer  the  questions 
that  follow  immediately. 
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*    Be  sure  you  understand  the  instructions.   When  in  doubt, 
ask  the  experimenter . 


***    You  are  now  ready  to  proceed  with  the  task    ** 
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Questions  to  be  answered  after  completing  part  1 


1.   Describe  (in  words,  equation,  etc)  what  decision  rule  you 
followed  in  making  your  estimates . 


2.    Distribute  100  points  among  the  three  variables  you  used 
for  reaching  your  overall  estimate  -  in  accordance  with  the 
importance  you  assigned  them  (be  sure  the  total  adds  up  to 
100)  . 

Variablel :      

Variable2:      

Variable3: 


TOTAL:  (out  of  100) 


3.  How  clear  were  the  instructions  regarding  the  task  in 
this  part  of  the  simulation? 

1      23456789 
Not  at  all  Very 

Clear  Clear 

4.  Please  try  to  describe  the  thinking  process  you  went 
through  in  making  your  predictions: 


***    END  OF  PART  1    *** 
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PART  2:  your  decisions 


INSTRUCTIONS 


*     You  are  about  to  start  part  two  of  the  simulation.   In 
this  part,  you  will  make  decisions  about  some  actual 
applicants  (i.e,  cases  drawn  from  resumes  of  real-life 
applicants) .   The  task  is  the  same  as  in  part  1:  rating 
applicants  on  a  scale  of  1-9. 


*  You  will  make  decisions  on  screening  applicants  for  three 
different  job  positions,  one  position  at  a  time.  There  will, 
therefore,  be  three  different  blocks  of  cases:  one  block  for 
each  position. 


*  Descriptions  of  the  individual  job  positions  are  not 
presented  to  you,  so  as  not  to  bias  your  decisions  (remember: 
your  task  is  to  emulate  the  committee,  i.e,  make  decisions  the 
committee  would  have  made)  .  However,  you  do  need  to  know  that 
the  committee  may  (or  may  not)  use  somewhat  different  decision 
rules  for  different  positions. 


*     Since  the  task  is  for  real  life  applicants,  the  system 
will  provide  you  with  decision  feedback  to  help  you  make 
better  decisions  (i.e.,  to  be  as  close  to  the  committee  scores 
as  possible) .   The  next  few  pages  explain  what  decision 
feedback  is  and  how  you  can  use  it. 


*     For  each  block  (i.e.,  each  job  position),  the  general 
sequence  is  as  follows: 

1.  You  are  given  a  set  of  2  6  cases. 

2.  You  make  estimates  on  these  cases,  in  any  order  you  like. 

The  up  and  down  arrows  can  be  used  for  moving  around  the  score 
windows  (you  can  change  your  scores  by  simply  typing  over  the 
old  score) . 

3.  If  you  press  the  END  key  at  any  point,  the  system  will 
provide  you  with  decision  feedback  (however,  you  need  to  enter 
at  least  10  scores  in  order  to  get  such  feedback) .   If,  after 
receiving  decision  feedback,  you  would  like  to  change  any 
score,  you  can  do  so. 
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4.    Once  you  have  entered  a  score,  you  can  get  information 
about  the  actual  value  (i.e.,  the  committee  decision)  at  any 
time,  also  by  pressing  the  END  key.   However,  once  you  have 
received  information  about  the  actual  value  for  a  candidate, 
you  cannot  change  your  score  for  that  candidate. 


*  Once  you  have  completed  a  block  (i.e.,  decided  on  all  26 
scores)  ,  press  the  HOME  key  to  move  to  the  next  block.  At  the 
end  of  each  block,  please  answer  the  questions  for  that  block 
on  the  questionnaire. 


*  At  every  stage,  the  bottom  row  of  the  screen  will  provide 
you  with  instructions  about  what  to  do  next . 

*  The  next  few  pages  explain  decision  feedback. 

*  If  you  have  a  question,  ask  the  experimenter. 
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Decision  Feedback 


*    What  is  Decision  Feedback? 

Decision  feedback  is  diagnostic  information  provided  by 
the  system  on  your  decision  processes  and  that  of  the 
committee .   You  can  access  this  feedback  when  making  your 
decisions  about  candidates.   (In  order  to  compute  the 
information,  however,  the  system  needs  at  least  10  scores  from 
you)  .   By  accessing  such  feedback,  decision  makers  can  derive 
better  insight  into  their  decisions  processes .   This  enables 
them  to  revise  and  improve  their  decisions  (or  scores,  in  this 
case)  through  a  what-if  mode  of  analysis . 


*  How  do  I  use  Decision  Feedback  in  Making  Decisions? 

Typically,  you  the  decision  maker,  would  use  the  feedback  as 
follows : 

1.  Make  some  tentative  decisions  (i.e.,  candidate  scores). 

2 .  Ask  the  system  for  feedback . 

3.  Refine  your  decisions  accordingly. 

You  may  do  this  1-3  sequence  within  a  block,  as  many  times 
(and  with  as  many  scores)  as  you  wish. 

*  Types  of  Decision  Feedback 

The  system  will  provide  you  with  the  following  5  types  of 
feedback : 

1 .    Information  on  your  decision  rule : 

Decision  makers  are  sometimes  unable  to  specify  precisely,  a 
particular  decision  rule  (in  this  case,  weights  assigned  to 
variables) .   The  system  will  track  the  weights  you  are  using 
(in  formulating  your  scores) ,  and  will  display  them  through  a 
stacked-bar  chart  in  figure  1 . 

SEE  FIGURE   2   CHAPTER   III   PAGE  26 

How  do  I  use  it? 

1 .   Make  sure  the  weights  displayed  are  actually  the  ones  you 
want  applied. 
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2.  If  not,  revise  your  scores,  and  see  how  the  weights 
change . 

3.  Iterate  between  1-2  till  the  system  shows  weights  you 
actually  want  applied. 


2.    Information  on  your  consistency: 

Sometimes,  after  decision  makers  have  specified  their  decision 
rule,  they  are  unable  to  apply  them  consistently.   The  system 
will  calculate  the  scores  you  would  have  given  had  you  been 
completely  consistent  with  your  decision  rule. 


How  do  I  use  it? 

1 .  Check  your  scores  against  the  consistency  scores . 

2.  Revise  your  scores  if  you  need  or  wish  to. 

3 .  Iterate  between  1-2  till  your  scores  match  with  or  are 
close  to  the  consistency  scores . 

3.  Information  on  the  committee's  decision  rule: 

Instead  of  trying  to  figure  out  from  several  examples  what 
rule  the  committee  is  using,  it  is  more  effective  if 

SEE  FIGURE  2   CHAPTER   III  PAGE  26 

How  do  I  use  it? 

1 .   Use  the  feedback  to  get  an  idea  of  what  decision  rule  the 
committee  has  been  following. 

4.  Information  on  the  committee's  decision  rule  and  yours 

This  is  actually  a  combination  of  feedback  1  and  feedback  3 . 
It  enables  you  to  compare  your  decision  rule  with  that  of  the 
committee  and  thereby  emulate  the  committee  better. 

SEE  FIGURE   3   CHAPTER   III  PAGE  21 
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How  do  I  use  it? 

1 .  Check  weights  you  have  given  versus  weights  given  by  the 
committee . 

2 .  Revise  your  scores  if  you  need  to . 

3 .  Iterate  between  1-2  till  your  weights  match  with  or  are 
close  to  the  committee  weights . 


5 .    Information  on  your  decision  rule  and  consistency 

This  is  actually  a  combination  of  feedback  1  and  feedback  2 . 
The  idea  here  is  to  let  you  revise  your  weights  without  losing 
your  consistency  at  the  same  time . 

How  do  I  use  it? 

1 .  Make  sure  the  weights  displayed  are  actually  the  ones  you 
want  applied. 

2.  If  not,  revise  your  scores,  and  see  how  the  weights 
change . 

3.  Check  your  scores  against  the  consistency  scores. 
2 .    Revise  your  scores  if  you  need  or  wish  to . 


*** 


You  are  now  ready  to  proceed  with  the  task    *** 
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Questions  to  be  answered  after  completing  BLOCK  1 


1.   Describe  (in  words,  equation,  etc)  what  decision  rule  you 
followed  in  making  your  own  estimates  (in  this  block) ? 


2.    Distribute  100  points  among  the  three  variables  you  used 
for  reaching  your  overall  estimate  -  in  accordance  with  the 
importance  you  assigned  them  (be  sure  the  total  adds  up  to 
100)  . 

Variablel:      

Variable2 :      

Variable3 : 


3 .    How  do  you  think  the  committee  weighted  the  three 
variables  in  this  block?   In  other  words,  distribute  100 
points  among  the  three  variables  according  to  how  you  think 
the  committee  did. 

Variablel:      

Variable2 :      

Variable3 : 


4.    In  this  block,  did  you  request  decision  feedback  at  any 
time  from  the  system  (Y/N) ?   


5.    If  YES,  try  to  describe  how  you  used  decision  feedback  in 
making  your  decisions . 


***    PLEASE  PROCEED  TO  BLOCK  2    *** 
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Questions  to  be  answered  after  completing  BLOCK  2 


1.   Describe  (in  words,  equation,  etc)  what  decision  rule  you 
followed  in  making  your  own  estimates  (in  this  block) ? 


2.    Distribute  100  points  among  the  three  variables  you  used 
for  reaching  your  overall  estimate  -  in  accordance  with  the 
importance  you  assigned  them  (be  sure  the  total  adds  up  to 
100)  . 

Variablel:      

Variable2:      

Variable3 : 


3 .    How  do  you  think  the  committee  weighted  the  three 
variables  in  this  block?   In  other  words,  distribute  100 
points  among  the  three  variables  according  to  how  you  think 
the  committee  did. 

Variablel:      

Variable2 :      

Variable3 : 


4.    In  this  block,  did  you  request  decision  feedback  at  any 
time  from  the  system  (Y/N) ?    


5.   If  YES,  try  to  describe  how  you  used  decision  feedback  in 
making  your  decisions . 


***    PLEASE  PROCEED  TO  BLOCK  3    *** 
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Questions  to  be  answered  after  completing  BLOCK  3 


1.   Describe  (in  words,  equation,  etc)  what  decision  rule  you 
followed  in  making  your  own  estimates  (in  this  block) ? 


2.    Distribute  100  points  among  the  three  variables  you  used 
for  reaching  your  overall  estimate  -  in  accordance  with  the 
importance  you  assigned  them  (be  sure  the  total  adds  up  to 
100)  . 

Variablel :      

Variable2 :      

Variable3 : 


3 .    How  do  you  think  the  committee  weighted  the  three 
variables  in  this  block?   In  other  words,  distribute  100 
points  among  the  three  variables  according  to  how  you  think 
the  committee  did. 

Variablel :      

Variable2:      

Variable3 : 


4.    In  this  block,  did  you  request  decision  feedback  at  any 
time  from  the  system  (Y/N) ?    


5.    If  YES,  try  to  describe  how  you  used  decision  feedback  in 
making  your  decisions . 


***    PLEASE  turn  over  for  further  instructions   *** 


66 


Questions  to  be  answered  after  completing  PART  2 


1.  To  what  extent  were  the  concepts  of  decision  feedback  (as 
explained  by  the  instructions  and  the  experimenter)  clear  to 
you? 

123456789 
Not   at   all  Very 

Clear  Clear 


2 .   To  what  extent  was  decision  feedback  helpful  in  improving 
your  own  decision? 

123456789 
Not  at  all  Very 

Helpful  Helpful 


3 .   To  what  extent  was  information  about  the  actual  committee 
scores  helpful  in  improving  your  own  decision? 

123456789 
Not  at  all  Very 

Helpful  Helpful 


4.    How  clear  were  the  instructions  regarding  the  task  in 
this  part  of  the  simulation? 

123456789 
Not  at  all  Very 

Clear  Clear 


5.    Now  that  you  have  completed  the  task,  can  you  think  of 
any  other  factor  (other  variables,  etc)  that  may  have 
influenced  you  in  making  your  decisions? 


***    END  OF  PART  2    *** 
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PART  3:  debriefing 


1.    Have  you,  in  the  past,  been  associated  with  applicant 
screening  before  (Y/N) ? 


2.    If  YES,  to  what  extent  was  the  task  similar  to  this 
simulation? 

123456789 
Not  at  all  Very 

Similar  Similar 


3 .    How  interesting  was  the  task  you  just  performed? 

123456789 
Not  at  all  Very 

Interesting  Interesting 


How  realistic,  in  your  opinion,  was  the  task? 

123456789 
Not  at  all  Very 

Realistic  Realistic 

Please  comment: 


5.    How  serious  were  you  in  performing  the  task? 

123456789 
Not  at  all  Very 

Serious  Serious 


7 .    How  clear  were  the  instructions  generally? 

123456789 
Not  at  all  Very 

Clear  Clear 
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8.    How  easy  was  the  system  to  use? 

123456789 
Not    at    all  Very 

Easy  Easy 


9.    Please  give  us  some  information  about  yourself  (in 
absolute  confidence.   At  no  time  will  your  name  appear  in  the 
results.   The  data  will  only  be  used  in  an  aggregate 
statistical  sense) . 

(a)  Curriculum  enrolled  in: 

(b)  Sex  

( c )  Age  


(d)  Fulltime  work  experience 

(in  years) 

(e)  How  long  ago  (in  years)  did 
you  complete  your 
undergraduate  education? 


(f)   How  familiar  are  you  with  computers,  generally? 

123456789 
Not  at  all  Very 

Familiar  Familiar 


(f)   How  many  hours  (per  week)  do  you  use  computers? 


10.   Your  general  comments  regarding  the  simulation: 


***    END  OF  SIMULATION    *** 
Thank  you  for  your  participation 
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